Incorporation of existing elements into new entities is a recurring theme in many areas of endeavor. Code reuse, whether specifically involving templates, procedures, functions, libraries, components and in some cases entire software works, has been practiced in various forms since the dawn of computing. Warnings against reinventing the wheel are well-known to engineers and software developers in general. Contemporary reuse of software components lives on, particularly in the form of add-ons, plugins, drivers, and various open source initiatives, each of these existing elements offering myriad modes of encapsulation or integration within new applications. More recently, the trend toward reuse in software contexts increased in popularity with the proliferation of open architectures that encourage the integration of third party components to operate within the framework some larger application. One characteristic of such component integration involves coherent interaction between the integrated components and integrating applications. In many cases, the incorporation is seamless, as the former components, once integrated, appear to form a native part of a given application.
Yet despite its many benefits, component reuse in general and integration of third-party content in particular often carry with them the original shortcomings of said component. Furthermore, the extent of said shortcomings may be known in certain cases and accordingly worked around. In other cases, these shortcomings remain largely or entirely unknown and are only discovered following subsequent integration within an application. The latter involves a transfer of risk and vulnerability to the party integrating or importing said components (in addition to users of said resulting software), as the latter components' stability is not guaranteed. Relatedly, the often ambiguous nature and extent of the instability is an ill-defined characteristic. This flows from the fact that lack of stability in third party components undermines the stability of our client's application overall.
A related consideration to software component reuse is the specific paradigm by which said reused components are actually integrated. In the case of applications that make use of numerous graphics and/or video components, one important integration paradigm is that of compositing window elements. Compositing, also called composition, involves combining various elements from different graphical sources or respectively superposing them. Such sources may consist of the graphical outputs from one or more computer applications. A computer system's compositing functionality, typically implemented via a compositing window manager disposed within said system's graphical section, aims to generate a resulting output or “scene” comprising output from various display applications whose contents appear as though all said graphical elements are part of the same scene.
Techniques to correctly determine the value of each pixel within a display require considerable processing resources. Notably, such techniques involve identifying the topmost element (i.e. the item having the highest Z-order value) among two or more overlapping two-dimensional objects, this process being somewhat iterative and complexified when multiple superposed transparent objects are present. Additional resources are typically required if rapid changes associated with at any one or more said graphical elements occur, with further resources required when alpha compositing considerations are relevant. Although existing system-level compositing functionalities provide to each application its own compositing engine, said engines do not typically intercommunicate. This further causes said techniques to be resource-intensive, since such rapid changes are typically associated with increased display activity. Accordingly, composition for regions of pixels where such increased display activity takes place must be done more frequently than for those regions—or even overall displays—where less or no activity occurs. The need for increased composition activity typically results in increased demand placed on the responsible compositing window manager. Of course, to satisfy such increased demand for composition, a related increase in resource requirements is necessary. The absence of such resources result in a limited compositing capability of a computer system's display, with negative repercussions on performance and overall user experience using one or more applications on said computer system. To avoid such performance losses, dedicated handling of compositing functionality by graphics hardware has become commonplace. In certain cases, some technologies use screen scraping in order to mix different rendering technologies, however these techniques lack the efficiency required for video streams.
A noteworthy case in which composition capability is called for occurs when a considerable portion of a display area is occupied by a video window. Superposition of additional visual entities atop said video window, such as digital on-screen graphics (e.g. for branding or identification purposes), or user interface component objects (such as playback controls) may be imagined. The result of compositing is typically necessary for each pixel within a display, and any subsequent transparency or alpha channel blending atop it needs to be done as well. This is why ideally, such compositing is typically left to the graphics section—typically a dedicated graphics card. While such “stacking” of graphical entities (e.g. windowed applications) might be desirable for enhanced functionality, it likewise makes a system correspondingly vulnerable to the shortcomings of composition arising from the lack of communication between of respective compositing engines and the (windows and/or graphical elements of) various applications.
In the case of an application comprising an array of video feeds received from a network of security cameras, an intense amount of processing is required for compositing each feed in addition to the additional graphical information typically superposed upon each respective feed. The present state of the art limits the amount of video feed data that may be composited and displayed with respective additional visual entities atop said feeds, all while a computer system retains an acceptable level of performance. In addition, the growing need for both higher definition images and higher framerate within many applications in which lower quality video (such as security monitoring) was previously acceptable have made the status quo unacceptable, particularly in light of lacking inter-process communication between compositing managers.
Concurrent with the need to overcome, ensure, and maintain proper compositing functionality for high quality video applications is the need for a video monitoring system to cooperate with possibly numerous third-party software components, particularly for use in decoding and rendering graphical elements. Moreover, abnormal operation is a recurrent concern for all components within a system and said abnormal operation is all the more a cause for concern in cases where key components may be provided by third parties. A component reuse model admitting third party components introduces vulnerabilities resulting from component reuse may be minor or fairly innocuous. Yet in extreme cases, one or more reused components may be ill-suited, malfunctioning, or incompatible with the application as a whole into which they are integrated that the latter may crash, be impaired, and/or or terminate suddenly with particularly devastating consequences.
Likewise, window compositing is known for being a particularly compute-intensive operation. This is particularly the case when multiple desktop components contribute to the final image to be displayed. Computing the final color or shade of a pixel is likewise a constituent part of the compositing process. Various schemes to improve the performance of desktop compositing have been proposed but many of these are subject to the vagaries of operating system implementations, which typically differ from version to version. Retained and immediate mode rendering and composition capabilities are possible using various schemes, both software and hardware. However, a problem experienced with current client applications is that like all desktop applications, retained mode rendering prevails. Painting and repainting various portions of a desktop—including attendant garbage collection—becomes processing intensive, as all pixels of an entire desktop must be validated for the correctness of their pixel values, composition and rendering engines must operate on all of said pixels, following which said composition is issued to the graphics card. Such operations typically bear an enormous processing cost and have a correspondingly negative effect on user experience, blocking video decoding and playback threads typically present on the same process as a client application.
The isolation of software components both for purposes of encouraging component modularity as well as for facilitating fault isolation in design is known in specific fields of computing. However, overlapping technical challenges from the respective technologies of system-level computing and user interface applications involved in displaying graphical data and/or handling rapidly increasing volumes of video data have prevented a contemporary extension of such principles to fields such as video monitoring where they are needed. Such is the purpose of the present application.